<!DOCTYPE html>


<html lang="en">
  

    <head>
      <meta charset="utf-8" />
        
      <meta
        name="viewport"
        content="width=device-width, initial-scale=1, maximum-scale=1"
      />
      <title>程序员代码下的许豪杰(技术篇) |  麦田麦穗</title>
  <meta name="generator" content="hexo-theme-ayer">
      
      <link rel="shortcut icon" href="/favicon.ico" />
       
<link rel="stylesheet" href="/dist/main.css">

      <link
        rel="stylesheet"
        href="https://cdn.jsdelivr.net/gh/Shen-Yu/cdn/css/remixicon.min.css"
      />
      
<link rel="stylesheet" href="/css/custom.css">
 
      <script src="https://cdn.jsdelivr.net/npm/pace-js@1.0.2/pace.min.js"></script>
       
 

      <link
        rel="stylesheet"
        href="https://cdn.jsdelivr.net/npm/@sweetalert2/theme-bulma@5.0.1/bulma.min.css"
      />
      <script src="https://cdn.jsdelivr.net/npm/sweetalert2@11.0.19/dist/sweetalert2.min.js"></script>

      <!-- mermaid -->
      
      <style>
        .swal2-styled.swal2-confirm {
          font-size: 1.6rem;
        }
      </style>
    </head>
  </html>
</html>


<body>
  <div id="app">
    
      
    <main class="content on">
      <section class="outer">
  <article
  id="post-python/程序员代码下的许豪杰-技术篇"
  class="article article-type-post"
  itemscope
  itemprop="blogPost"
  data-scroll-reveal
>
  <div class="article-inner">
    
    <header class="article-header">
       
<h1 class="article-title sea-center" style="border-left:0" itemprop="name">
  程序员代码下的许豪杰(技术篇)
</h1>
 

      
    </header>
     
    <div class="article-meta">
      <a href="/2017/08/05/python/%E7%A8%8B%E5%BA%8F%E5%91%98%E4%BB%A3%E7%A0%81%E4%B8%8B%E7%9A%84%E8%AE%B8%E8%B1%AA%E6%9D%B0-%E6%8A%80%E6%9C%AF%E7%AF%87/" class="article-date">
  <time datetime="2017-08-05T05:16:00.000Z" itemprop="datePublished">2017-08-05</time>
</a>   
<div class="word_count">
    <span class="post-time">
        <span class="post-meta-item-icon">
            <i class="ri-quill-pen-line"></i>
            <span class="post-meta-item-text"> Word count:</span>
            <span class="post-count">1.8k</span>
        </span>
    </span>

    <span class="post-time">
        &nbsp; | &nbsp;
        <span class="post-meta-item-icon">
            <i class="ri-book-open-line"></i>
            <span class="post-meta-item-text"> Reading time≈</span>
            <span class="post-count">7 min</span>
        </span>
    </span>
</div>
 
    </div>
      
    <div class="tocbot"></div>




  
    <div class="article-entry" itemprop="articleBody">
       
  <blockquote>
<p>接上篇，这一篇将从技术层面讲讲是如何实现的。阅读本文您将会了解如何用python爬取微博的评论以及如何用python word_cloud库进行数据可视化。</p>
</blockquote>
<p>上一篇:<a target="_blank" rel="noopener" href="http://www.jianshu.com/p/ab49113c1fda">程序员代码下的许豪杰</a></p>
<span id="more"></span>

<h2 id="准备工作"><a href="#准备工作" class="headerlink" title="准备工作"></a>准备工作</h2><p>打开微博pc m站并找到许豪杰该条微博地址:<a target="_blank" rel="noopener" href="https://m.weibo.cn/status/4132385564040383">https://m.weibo.cn/status/4132385564040383</a></p>
<p><strong>为什么要用m站地址？因为m站可以直接抓取到api json数据,而pc站虽然也有api返回的是html,相比而言选取m站会省去很多麻烦</strong></p>
<p>打开该页面，并且用chrome 的检查工具 查看network，可以获取到评论的api地址。</p>
<p><img src="http://upload-images.jianshu.io/upload_images/4033700-382e5a613595bc1c.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="chrome查看"></p>
<h2 id="数据抓取"><a href="#数据抓取" class="headerlink" title="数据抓取"></a>数据抓取</h2><p>首先观察api返回</p>
<p><img src="http://upload-images.jianshu.io/upload_images/4033700-f2aa3ddad169fe31.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="image.png"></p>
<p>从返回地址上可以看到可以通过参数page 改变请求的页码,并且每页都回返回总条数和总页码数。这里我决定采用<strong>多线程来抓去</strong>(其实数据量不大,也可以单线程跑)。</p>
<h3 id="其中在爬取数据的时候会面临几个问题："><a href="#其中在爬取数据的时候会面临几个问题：" class="headerlink" title="其中在爬取数据的时候会面临几个问题："></a>其中在爬取数据的时候会面临几个问题：</h3><p><strong>1.存储选择</strong></p>
<p>我这里选用了MongoDB作为数据存储，因为api通常返回的是json数据而json结构和MongoDB的存储方式可以结合的很默契，不需要经过任何处理可以直接的进行插入。</p>
<p><strong>2.防爬虫</strong></p>
<p>很多网站可能会做一些防爬虫的处理，面对同一个请求ip的短时间的高频率请求会进行服务隔断(直接告诉你服务不可用)，这个时候可以去网上找一些代理进行请求。</p>
<p><strong>3.多线程的任务分配</strong></p>
<p>采用多线程爬取你当然不能让多个线程去爬取同样的链接做别人已经做过的事情，那样多线程毫无意义。所以你需要制定一套规则，让不同线程爬取不同的链接。</p>
<pre class="language-python" data-language="python"><code class="language-python"><span class="token comment"># coding=utf-8</span>
<span class="token keyword">from</span> __future__ <span class="token keyword">import</span> division
<span class="token keyword">from</span> pymongo <span class="token keyword">import</span> MongoClient
<span class="token keyword">import</span> requests
<span class="token keyword">import</span> sys
<span class="token keyword">import</span> re
<span class="token keyword">import</span> random
<span class="token keyword">import</span> time
<span class="token keyword">import</span> logging
<span class="token keyword">import</span> threading
<span class="token keyword">import</span> json
<span class="token keyword">from</span> os <span class="token keyword">import</span> path
<span class="token keyword">import</span> math

<span class="token comment"># 爬取微博评论</span>
<span class="token comment"># m站微博地址</span>
weibo_url <span class="token operator">=</span> <span class="token string">'https://m.weibo.cn/status/4132385564040383'</span> 

thread_nums <span class="token operator">=</span> <span class="token number">5</span>  <span class="token comment">#线程数</span>

<span class="token comment">#代理地址</span>
proxies <span class="token operator">=</span> <span class="token punctuation">&#123;</span>
  <span class="token string">"http"</span><span class="token punctuation">:</span> <span class="token string">"http://171.92.4.67:9000"</span><span class="token punctuation">,</span>
  <span class="token string">"http"</span><span class="token punctuation">:</span> <span class="token string">"http://163.125.222.240:8118"</span><span class="token punctuation">,</span>
  <span class="token string">"http"</span><span class="token punctuation">:</span> <span class="token string">"http://121.232.145.251:9000"</span><span class="token punctuation">,</span>
  <span class="token string">"http"</span><span class="token punctuation">:</span> <span class="token string">"http://121.232.147.247:9000"</span><span class="token punctuation">,</span>
  
<span class="token punctuation">&#125;</span>


<span class="token comment"># 创建 日志 对象</span>
logger <span class="token operator">=</span> logging<span class="token punctuation">.</span>getLogger<span class="token punctuation">(</span><span class="token punctuation">)</span>
handler <span class="token operator">=</span> logging<span class="token punctuation">.</span>StreamHandler<span class="token punctuation">(</span><span class="token punctuation">)</span>
formatter <span class="token operator">=</span> logging<span class="token punctuation">.</span>Formatter<span class="token punctuation">(</span>
    <span class="token string">'%(asctime)s %(name)-12s %(levelname)-8s %(message)s'</span><span class="token punctuation">)</span>
handler<span class="token punctuation">.</span>setFormatter<span class="token punctuation">(</span>formatter<span class="token punctuation">)</span>
logger<span class="token punctuation">.</span>addHandler<span class="token punctuation">(</span>handler<span class="token punctuation">)</span>
logger<span class="token punctuation">.</span>setLevel<span class="token punctuation">(</span>logging<span class="token punctuation">.</span>DEBUG<span class="token punctuation">)</span>

mongoconn <span class="token operator">=</span> MongoClient<span class="token punctuation">(</span><span class="token string">'127.0.0.1'</span><span class="token punctuation">,</span> <span class="token number">27017</span><span class="token punctuation">)</span>
mdb <span class="token operator">=</span> mongoconn<span class="token punctuation">.</span>data_analysis
das_collection <span class="token operator">=</span> mdb<span class="token punctuation">.</span>weibo

weiboid_reobj <span class="token operator">=</span> re<span class="token punctuation">.</span>match<span class="token punctuation">(</span><span class="token string">r'.*status/(\d+)'</span><span class="token punctuation">,</span> weibo_url<span class="token punctuation">)</span>
weibo_id <span class="token operator">=</span> weiboid_reobj<span class="token punctuation">.</span>group<span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span>


<span class="token keyword">def</span> <span class="token function">scrapy_comments</span><span class="token punctuation">(</span>weibo_id<span class="token punctuation">,</span> page<span class="token punctuation">)</span><span class="token punctuation">:</span>
    weibo_comment_url <span class="token operator">=</span> <span class="token string">'https://m.weibo.cn/api/comments/show?id=%s&amp;page=%d'</span> <span class="token operator">%</span> <span class="token punctuation">(</span>
        weibo_id<span class="token punctuation">,</span> page<span class="token punctuation">)</span>
    res <span class="token operator">=</span> requests<span class="token punctuation">.</span>get<span class="token punctuation">(</span>weibo_comment_url<span class="token punctuation">)</span>
    res_obj <span class="token operator">=</span> json<span class="token punctuation">.</span>loads<span class="token punctuation">(</span>res<span class="token punctuation">.</span>content<span class="token punctuation">)</span>
    <span class="token keyword">return</span> res_obj


<span class="token keyword">def</span> <span class="token function">import_comments</span><span class="token punctuation">(</span>threadName<span class="token punctuation">,</span> weibo_id<span class="token punctuation">,</span> page_start<span class="token punctuation">,</span> page_end<span class="token punctuation">)</span><span class="token punctuation">:</span>
    logger<span class="token punctuation">.</span>info<span class="token punctuation">(</span><span class="token string">'开始线程:%s'</span> <span class="token operator">%</span> threadName<span class="token punctuation">)</span>
    <span class="token keyword">for</span> page <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span>page_start<span class="token punctuation">,</span> page_end <span class="token operator">+</span> <span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
        logging<span class="token punctuation">.</span>info<span class="token punctuation">(</span><span class="token string">'读取第%s页'</span> <span class="token operator">%</span> page<span class="token punctuation">)</span>
        time<span class="token punctuation">.</span>sleep<span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span>
        <span class="token comment"># continue</span>
        <span class="token keyword">try</span><span class="token punctuation">:</span>
            res_obj <span class="token operator">=</span> scrapy_comments<span class="token punctuation">(</span>weibo_id<span class="token punctuation">,</span> page<span class="token punctuation">)</span>
            logging<span class="token punctuation">.</span>info<span class="token punctuation">(</span><span class="token string">'该页有%s条记录'</span> <span class="token operator">%</span> <span class="token builtin">len</span><span class="token punctuation">(</span>res_obj<span class="token punctuation">[</span><span class="token string">'data'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
        <span class="token keyword">except</span><span class="token punctuation">:</span>
            logging<span class="token punctuation">.</span>error<span class="token punctuation">(</span><span class="token string">'读取%s页时发生错误'</span> <span class="token operator">%</span> page<span class="token punctuation">)</span>
            <span class="token keyword">continue</span>
        <span class="token keyword">if</span> res_obj<span class="token punctuation">[</span><span class="token string">'ok'</span><span class="token punctuation">]</span> <span class="token operator">==</span> <span class="token number">1</span><span class="token punctuation">:</span>
            comments <span class="token operator">=</span> res_obj<span class="token punctuation">[</span><span class="token string">'data'</span><span class="token punctuation">]</span>
            <span class="token keyword">for</span> comment <span class="token keyword">in</span> comments<span class="token punctuation">:</span>
                comment_text <span class="token operator">=</span> re<span class="token punctuation">.</span>sub<span class="token punctuation">(</span>
                    <span class="token string">r'&lt;/?\w+[^>]*>'</span><span class="token punctuation">,</span> <span class="token string">''</span><span class="token punctuation">,</span> comment<span class="token punctuation">[</span><span class="token string">'text'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">.</span>encode<span class="token punctuation">(</span><span class="token string">'utf-8'</span><span class="token punctuation">)</span>
                <span class="token keyword">if</span> re<span class="token punctuation">.</span>search<span class="token punctuation">(</span><span class="token string">r'回复@.*:'</span><span class="token punctuation">,</span> comment_text<span class="token punctuation">)</span><span class="token punctuation">:</span>
                    <span class="token comment"># 过滤掉回复别人的评论</span>
                    <span class="token keyword">continue</span>
                comment<span class="token punctuation">[</span><span class="token string">'text'</span><span class="token punctuation">]</span> <span class="token operator">=</span> comment_text
                comment<span class="token punctuation">[</span><span class="token string">'weibo_id'</span><span class="token punctuation">]</span> <span class="token operator">=</span> weibo_id
                logging<span class="token punctuation">.</span>info<span class="token punctuation">(</span><span class="token string">'读取评论:%s'</span> <span class="token operator">%</span> comment<span class="token punctuation">[</span><span class="token string">'id'</span><span class="token punctuation">]</span><span class="token punctuation">)</span>
                <span class="token keyword">try</span><span class="token punctuation">:</span>
                    <span class="token keyword">if</span> das_collection<span class="token punctuation">.</span>find_one<span class="token punctuation">(</span><span class="token punctuation">&#123;</span><span class="token string">'id'</span><span class="token punctuation">:</span> comment<span class="token punctuation">[</span><span class="token string">'id'</span><span class="token punctuation">]</span><span class="token punctuation">&#125;</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
                        logging<span class="token punctuation">.</span>info<span class="token punctuation">(</span><span class="token string">'在mongodb中存在'</span><span class="token punctuation">)</span>
                    <span class="token keyword">else</span><span class="token punctuation">:</span>
                        logging<span class="token punctuation">.</span>info<span class="token punctuation">(</span><span class="token string">'插入记录:%s'</span> <span class="token operator">%</span> comment<span class="token punctuation">[</span><span class="token string">'id'</span><span class="token punctuation">]</span><span class="token punctuation">)</span>
                        das_collection<span class="token punctuation">.</span>insert_one<span class="token punctuation">(</span>comment<span class="token punctuation">)</span>
                <span class="token keyword">except</span><span class="token punctuation">:</span>
                    logging<span class="token punctuation">.</span>error<span class="token punctuation">(</span><span class="token string">'mongodb发生错误'</span><span class="token punctuation">)</span>
        <span class="token keyword">else</span><span class="token punctuation">:</span>
            logging<span class="token punctuation">.</span>error<span class="token punctuation">(</span><span class="token string">'读取第%s页时发生错误'</span> <span class="token operator">%</span> page<span class="token punctuation">)</span>
    logging<span class="token punctuation">.</span>info<span class="token punctuation">(</span><span class="token string">'线程%s结束'</span> <span class="token operator">%</span> threadName<span class="token punctuation">)</span>
    <span class="token comment"># res_obj = scrapy_comments(weibo_id, page)</span>


<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">'__main__'</span><span class="token punctuation">:</span>
    <span class="token comment"># 分配不同链接到不同的线程上去</span>
    res_obj <span class="token operator">=</span> scrapy_comments<span class="token punctuation">(</span>weibo_id<span class="token punctuation">,</span> <span class="token number">1</span><span class="token punctuation">)</span>
    <span class="token keyword">if</span> res_obj<span class="token punctuation">[</span><span class="token string">'ok'</span><span class="token punctuation">]</span> <span class="token operator">==</span> <span class="token number">1</span><span class="token punctuation">:</span>
        total_number <span class="token operator">=</span> res_obj<span class="token punctuation">[</span><span class="token string">'total_number'</span><span class="token punctuation">]</span>
        logging<span class="token punctuation">.</span>info<span class="token punctuation">(</span><span class="token string">'该条微博有:%s条评论'</span> <span class="token operator">%</span> total_number<span class="token punctuation">)</span>
        max_page <span class="token operator">=</span> res_obj<span class="token punctuation">[</span><span class="token string">'max'</span><span class="token punctuation">]</span>
        page_nums <span class="token operator">=</span> math<span class="token punctuation">.</span>ceil<span class="token punctuation">(</span>max_page <span class="token operator">/</span> thread_nums<span class="token punctuation">)</span>
    <span class="token keyword">else</span><span class="token punctuation">:</span>
        <span class="token keyword">raise</span>

    <span class="token comment"># print max_page</span>
    <span class="token comment"># print page_nums</span>

    <span class="token keyword">for</span> i <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">,</span> thread_nums <span class="token operator">+</span> <span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
        <span class="token keyword">if</span> i <span class="token operator">&lt;</span> thread_nums<span class="token punctuation">:</span>
            page_end <span class="token operator">=</span> page_nums <span class="token operator">*</span> i
        <span class="token keyword">else</span><span class="token punctuation">:</span>
            page_end <span class="token operator">=</span> max_page
        page_start <span class="token operator">=</span> <span class="token punctuation">(</span>i <span class="token operator">-</span> <span class="token number">1</span><span class="token punctuation">)</span> <span class="token operator">*</span> page_nums <span class="token operator">+</span> <span class="token number">1</span>

        t <span class="token operator">=</span> threading<span class="token punctuation">.</span>Thread<span class="token punctuation">(</span>target<span class="token operator">=</span>import_comments<span class="token punctuation">,</span> args<span class="token operator">=</span><span class="token punctuation">(</span>
            i<span class="token punctuation">,</span> weibo_id<span class="token punctuation">,</span> <span class="token builtin">int</span><span class="token punctuation">(</span>page_start<span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token builtin">int</span><span class="token punctuation">(</span>page_end<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
        t<span class="token punctuation">.</span>start<span class="token punctuation">(</span><span class="token punctuation">)</span>
</code></pre>
<h2 id="数据整理可视化-data-visualization"><a href="#数据整理可视化-data-visualization" class="headerlink" title="数据整理可视化(data visualization)"></a>数据整理可视化(data visualization)</h2><p>运行脚本完毕，我的MongoDB得到了2万多条评论数据，接下来要做的事是对这部分数据进行提取、清洗、结构化等操作。这里顺便说明一下python 数据分析的 大致基本流程。</p>
<p><strong>1.与外界进行交互</strong><br>这个过程包括数据的获取、读取。不管是从网络资源上爬取、还是从现有资源(各样的文件如文本、excel、数据库存储对象)</p>
<p><strong>2.准备工作</strong><br>对数据进行清洗(cleaning)、修整(munging)、整合(combining)、规范化(normalizing)、重塑(reshaping)、切片(slicing)和切块(dicing)</p>
<p><strong>3.转换</strong><br>对数据集做一些数学和统计运算产生新的数据集</p>
<p><strong>4.建模和计算</strong><br>将数据跟统计模型、机器学习算法或其他计算工具联系起来</p>
<p><strong>5.展示</strong><br>创建交互式的或静态的图片或文字摘要</p>
<p>下面我们来进行2、3及5的工作:</p>
<pre class="language-python" data-language="python"><code class="language-python"><span class="token comment"># coding=utf-8</span>
<span class="token keyword">import</span> sys
<span class="token keyword">from</span> pymongo <span class="token keyword">import</span> MongoClient
<span class="token keyword">import</span> random
<span class="token comment"># 分词库</span>
<span class="token comment"># from snownlp import SnowNLP</span>
<span class="token keyword">import</span> jieba
<span class="token keyword">import</span> uniout
<span class="token keyword">from</span> collections <span class="token keyword">import</span> Counter<span class="token punctuation">,</span> OrderedDict
<span class="token comment"># 词语云 文本统计可视化库</span>
<span class="token keyword">from</span> wordcloud <span class="token keyword">import</span> WordCloud


mongoconn <span class="token operator">=</span> MongoClient<span class="token punctuation">(</span><span class="token string">'127.0.0.1'</span><span class="token punctuation">,</span> <span class="token number">27017</span><span class="token punctuation">)</span>
mdb <span class="token operator">=</span> mongoconn<span class="token punctuation">.</span>data_analysis
das_collection <span class="token operator">=</span> mdb<span class="token punctuation">.</span>weibo


total_counts <span class="token operator">=</span> das_collection<span class="token punctuation">.</span>find<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>count<span class="token punctuation">(</span><span class="token punctuation">)</span>

<span class="token comment"># random_int = random.randint(0, total_counts - 1)</span>
docs <span class="token operator">=</span> das_collection<span class="token punctuation">.</span>find<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">print</span> docs<span class="token punctuation">.</span>count<span class="token punctuation">(</span><span class="token punctuation">)</span>
words_counts <span class="token operator">=</span> <span class="token punctuation">&#123;</span><span class="token punctuation">&#125;</span>
<span class="token keyword">for</span> doc <span class="token keyword">in</span> docs<span class="token punctuation">:</span>
    <span class="token keyword">print</span> doc
    comment_text <span class="token operator">=</span> doc<span class="token punctuation">[</span><span class="token string">'text'</span><span class="token punctuation">]</span><span class="token punctuation">.</span>encode<span class="token punctuation">(</span><span class="token string">'utf-8'</span><span class="token punctuation">)</span>
    <span class="token keyword">if</span> <span class="token builtin">len</span><span class="token punctuation">(</span>comment_text<span class="token punctuation">)</span> <span class="token operator">==</span> <span class="token number">0</span><span class="token punctuation">:</span>
        <span class="token keyword">continue</span>
    words <span class="token operator">=</span> jieba<span class="token punctuation">.</span>cut<span class="token punctuation">(</span>comment_text<span class="token punctuation">)</span>
    <span class="token keyword">for</span> word <span class="token keyword">in</span> words<span class="token punctuation">:</span>
        <span class="token keyword">if</span> word <span class="token keyword">not</span> <span class="token keyword">in</span> words_counts<span class="token punctuation">:</span>
            words_counts<span class="token punctuation">[</span>word<span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token number">1</span>
        <span class="token keyword">else</span><span class="token punctuation">:</span>
            words_counts<span class="token punctuation">[</span>word<span class="token punctuation">]</span> <span class="token operator">+=</span> <span class="token number">1</span>

<span class="token keyword">for</span> word <span class="token keyword">in</span> words_counts<span class="token punctuation">.</span>keys<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
    <span class="token keyword">if</span> words_counts<span class="token punctuation">[</span>word<span class="token punctuation">]</span> <span class="token operator">&lt;</span> <span class="token number">2</span> <span class="token keyword">or</span> <span class="token builtin">len</span><span class="token punctuation">(</span>word<span class="token punctuation">)</span> <span class="token operator">&lt;</span> <span class="token number">2</span><span class="token punctuation">:</span>
        <span class="token keyword">del</span> words_counts<span class="token punctuation">[</span>word<span class="token punctuation">]</span>

<span class="token comment"># print words_counts.items()</span>
<span class="token comment">#注意要让中文不乱码要指定中文字体</span>
<span class="token comment">#fit_words 接收参数是dict  eg:&#123;'你':333,'好':23&#125;  文字:出现次数</span>
wordcloud <span class="token operator">=</span> WordCloud<span class="token punctuation">(</span>
    font_path<span class="token operator">=</span><span class="token string">'/Users/cwp/font/msyh.ttf'</span><span class="token punctuation">,</span>
    background_color<span class="token operator">=</span><span class="token string">'white'</span><span class="token punctuation">,</span>
    width<span class="token operator">=</span><span class="token number">1200</span><span class="token punctuation">,</span>
    height<span class="token operator">=</span><span class="token number">1000</span>
<span class="token punctuation">)</span><span class="token punctuation">.</span>fit_words<span class="token punctuation">(</span>words_counts<span class="token punctuation">)</span>
<span class="token keyword">import</span> matplotlib<span class="token punctuation">.</span>pyplot <span class="token keyword">as</span> plt
plt<span class="token punctuation">.</span>imshow<span class="token punctuation">(</span>wordcloud<span class="token punctuation">,</span> interpolation<span class="token operator">=</span><span class="token string">'bilinear'</span><span class="token punctuation">)</span>
plt<span class="token punctuation">.</span>axis<span class="token punctuation">(</span><span class="token string">"off"</span><span class="token punctuation">)</span>
plt<span class="token punctuation">.</span>show<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>介绍下以上代码：<br>我们主要用到了2个工具，jieba和word_cloud。前者对中文进行分词后者图形化展示词语的出现频率。<br>众所周知，中文系的语言处理恐怕是最难的自然语言处理(NLP)的语种。就基本的分词而言都是一项比较困难的工作,(英语句子中每个单词都是有空格分开的，而中文是由单个字组成词连接成串组成句).<br>举个例子,请用“孩提”造句,”那个男孩提交完代码就下班了”。如果人工分词，可以知道”男孩”和”提交”应该是分开的2个词，<strong>但是对于机器而言，要辨别”提”应该与”男”还是”交”进行组词就很难办了</strong>。要想机器能够更精确的辨别这类问题，就需要让机器不停学习，让它知道这种情况该这么分而不是那么分。研究中文自然语言处理将是一个长久而大的工程，对于分析数据(我们不是要研究自然语言处理😏)，这里就借助jieba这个库进行工作了.</p>
<p>对于word_cloud,图形化文本统计，网上有不少的博文都贴了代码，但我想说的是我不了解它们是不是真的运行出了结果。因为fit_words 这个函数接收的是dict而不是list，官方文档和函数doc其实写错了,在github上有披露。</p>
<p>最后得到结果:</p>
<p><img src="http://upload-images.jianshu.io/upload_images/4033700-dfbee241840ea23e.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="结果"></p>
<h2 id="一些用到的工具"><a href="#一些用到的工具" class="headerlink" title="一些用到的工具"></a>一些用到的工具</h2><p>1.<a target="_blank" rel="noopener" href="https://github.com/amueller/word_cloud">word_cloud A little word cloud generator in Python</a></p>
<p>2.<a target="_blank" rel="noopener" href="https://github.com/fxsjy/jieba">jieba 结巴中文分词</a></p>
<p>3.<a target="_blank" rel="noopener" href="http://docs.python-requests.org/en/master/">Requests is the only Non-GMO HTTP library for Python, safe for human consumption.</a></p>
 
      <!-- reward -->
      
      <div id="reword-out">
        <div id="reward-btn">
          Donate
        </div>
      </div>
      
    </div>
    

    <!-- copyright -->
    
    <div class="declare">
      <ul class="post-copyright">
        <li>
          <i class="ri-copyright-line"></i>
          <strong>Copyright： </strong>
          
          Copyright is owned by the author. For commercial reprints, please contact the author for authorization. For non-commercial reprints, please indicate the source.
          
        </li>
      </ul>
    </div>
    
    <footer class="article-footer">
       
<div class="share-btn">
      <span class="share-sns share-outer">
        <i class="ri-share-forward-line"></i>
        分享
      </span>
      <div class="share-wrap">
        <i class="arrow"></i>
        <div class="share-icons">
          
          <a class="weibo share-sns" href="javascript:;" data-type="weibo">
            <i class="ri-weibo-fill"></i>
          </a>
          <a class="weixin share-sns wxFab" href="javascript:;" data-type="weixin">
            <i class="ri-wechat-fill"></i>
          </a>
          <a class="qq share-sns" href="javascript:;" data-type="qq">
            <i class="ri-qq-fill"></i>
          </a>
          <a class="douban share-sns" href="javascript:;" data-type="douban">
            <i class="ri-douban-line"></i>
          </a>
          <!-- <a class="qzone share-sns" href="javascript:;" data-type="qzone">
            <i class="icon icon-qzone"></i>
          </a> -->
          
          <a class="facebook share-sns" href="javascript:;" data-type="facebook">
            <i class="ri-facebook-circle-fill"></i>
          </a>
          <a class="twitter share-sns" href="javascript:;" data-type="twitter">
            <i class="ri-twitter-fill"></i>
          </a>
          <a class="google share-sns" href="javascript:;" data-type="google">
            <i class="ri-google-fill"></i>
          </a>
        </div>
      </div>
</div>

<div class="wx-share-modal">
    <a class="modal-close" href="javascript:;"><i class="ri-close-circle-line"></i></a>
    <p>扫一扫，分享到微信</p>
    <div class="wx-qrcode">
      <img src="//api.qrserver.com/v1/create-qr-code/?size=150x150&data=http://visonforcoding.xyz/2017/08/05/python/%E7%A8%8B%E5%BA%8F%E5%91%98%E4%BB%A3%E7%A0%81%E4%B8%8B%E7%9A%84%E8%AE%B8%E8%B1%AA%E6%9D%B0-%E6%8A%80%E6%9C%AF%E7%AF%87/" alt="微信分享二维码">
    </div>
</div>

<div id="share-mask"></div>  
    </footer>
  </div>

   
  <nav class="article-nav">
    
      <a href="/2017/12/01/%E5%9C%B0%E7%90%86%E4%BD%8D%E7%BD%AE%E5%A4%84%E7%90%86%E4%B9%8Bmongodb/" class="article-nav-link">
        <strong class="article-nav-caption">上一篇</strong>
        <div class="article-nav-title">
          
            数据|地理位置geo处理之mongodb geo 索引
          
        </div>
      </a>
    
    
  </nav>

   
<!-- valine评论 -->
<div id="vcomments-box">
  <div id="vcomments"></div>
</div>
<script src="//cdn1.lncld.net/static/js/3.0.4/av-min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/valine@1.4.14/dist/Valine.min.js"></script>
<script>
  new Valine({
    el: "#vcomments",
    app_id: "",
    app_key: "",
    path: window.location.pathname,
    avatar: "monsterid",
    placeholder: "给我的文章加点评论吧~",
    recordIP: true,
  });
  const infoEle = document.querySelector("#vcomments .info");
  if (infoEle && infoEle.childNodes && infoEle.childNodes.length > 0) {
    infoEle.childNodes.forEach(function (item) {
      item.parentNode.removeChild(item);
    });
  }
</script>
<style>
  #vcomments-box {
    padding: 5px 30px;
  }

  @media screen and (max-width: 800px) {
    #vcomments-box {
      padding: 5px 0px;
    }
  }

  #vcomments-box #vcomments {
    background-color: #fff;
  }

  .v .vlist .vcard .vh {
    padding-right: 20px;
  }

  .v .vlist .vcard {
    padding-left: 10px;
  }
</style>

 
   
     
</article>

</section>
      <footer class="footer">
  <div class="outer">
    <ul>
      <li>
        Copyrights &copy;
        2015-2022
        <i class="ri-heart-fill heart_icon"></i> vison
      </li>
    </ul>
    <ul>
      <li>
        
      </li>
    </ul>
    <ul>
      <li>
        
        
        <span>
  <span><i class="ri-user-3-fill"></i>Visitors:<span id="busuanzi_value_site_uv"></span></span>
  <span class="division">|</span>
  <span><i class="ri-eye-fill"></i>Views:<span id="busuanzi_value_page_pv"></span></span>
</span>
        
      </li>
    </ul>
    <ul>
      
    </ul>
    <ul>
      
    </ul>
    <ul>
      <li>
        <!-- cnzz统计 -->
        
        <script type="text/javascript" src='https://s9.cnzz.com/z_stat.php?id=1278069914&amp;web_id=1278069914'></script>
        
      </li>
    </ul>
  </div>
</footer>    
    </main>
    <div class="float_btns">
      <div class="totop" id="totop">
  <i class="ri-arrow-up-line"></i>
</div>

    </div>
    <aside class="sidebar on">
      <button class="navbar-toggle"></button>
<nav class="navbar">
  
  <div class="logo">
    <a href="/"><img src="/images/ayer-side.svg" alt="麦田麦穗"></a>
  </div>
  
  <ul class="nav nav-main">
    
    <li class="nav-item">
      <a class="nav-item-link" href="/">主页</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/archives">归档</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/categories">分类</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/tags">标签</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/tags/%E6%97%85%E8%A1%8C/">旅行</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" target="_blank" rel="noopener" href="http://shenyu-vip.lofter.com">摄影</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/friends">友链</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/about">关于我</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/null">Gallery</a>
    </li>
    
  </ul>
</nav>
<nav class="navbar navbar-bottom">
  <ul class="nav">
    <li class="nav-item">
      
      <a class="nav-item-link nav-item-search"  title="Search">
        <i class="ri-search-line"></i>
      </a>
      
      
      <a class="nav-item-link" target="_blank" href="/atom.xml" title="RSS Feed">
        <i class="ri-rss-line"></i>
      </a>
      
    </li>
  </ul>
</nav>
<div class="search-form-wrap">
  <div class="local-search local-search-plugin">
  <input type="search" id="local-search-input" class="local-search-input" placeholder="Search...">
  <div id="local-search-result" class="local-search-result"></div>
</div>
</div>
    </aside>
    <div id="mask"></div>

<!-- #reward -->
<div id="reward">
  <span class="close"><i class="ri-close-line"></i></span>
  <p class="reward-p"><i class="ri-cup-line"></i>Buy me a cup of coffee~</p>
  <div class="reward-box">
    
    <div class="reward-item">
      <img class="reward-img" src="/images/alipay.jpeg">
      <span class="reward-type">支付宝</span>
    </div>
    
    
    <div class="reward-item">
      <img class="reward-img" src="/images/wallet.png">
      <span class="reward-type">微信</span>
    </div>
    
  </div>
</div>
    
<script src="/js/jquery-3.6.0.min.js"></script>
 
<script src="/js/lazyload.min.js"></script>

<!-- Tocbot -->
 
<script src="/js/tocbot.min.js"></script>

<script>
  tocbot.init({
    tocSelector: ".tocbot",
    contentSelector: ".article-entry",
    headingSelector: "h1, h2, h3, h4, h5, h6",
    hasInnerContainers: true,
    scrollSmooth: true,
    scrollContainer: "main",
    positionFixedSelector: ".tocbot",
    positionFixedClass: "is-position-fixed",
    fixedSidebarOffset: "auto",
  });
</script>

<script src="https://cdn.jsdelivr.net/npm/jquery-modal@0.9.2/jquery.modal.min.js"></script>
<link
  rel="stylesheet"
  href="https://cdn.jsdelivr.net/npm/jquery-modal@0.9.2/jquery.modal.min.css"
/>
<script src="https://cdn.jsdelivr.net/npm/justifiedGallery@3.7.0/dist/js/jquery.justifiedGallery.min.js"></script>

<script src="/dist/main.js"></script>

<!-- ImageViewer -->
 <!-- Root element of PhotoSwipe. Must have class pswp. -->
<div class="pswp" tabindex="-1" role="dialog" aria-hidden="true">

    <!-- Background of PhotoSwipe. 
         It's a separate element as animating opacity is faster than rgba(). -->
    <div class="pswp__bg"></div>

    <!-- Slides wrapper with overflow:hidden. -->
    <div class="pswp__scroll-wrap">

        <!-- Container that holds slides. 
            PhotoSwipe keeps only 3 of them in the DOM to save memory.
            Don't modify these 3 pswp__item elements, data is added later on. -->
        <div class="pswp__container">
            <div class="pswp__item"></div>
            <div class="pswp__item"></div>
            <div class="pswp__item"></div>
        </div>

        <!-- Default (PhotoSwipeUI_Default) interface on top of sliding area. Can be changed. -->
        <div class="pswp__ui pswp__ui--hidden">

            <div class="pswp__top-bar">

                <!--  Controls are self-explanatory. Order can be changed. -->

                <div class="pswp__counter"></div>

                <button class="pswp__button pswp__button--close" title="Close (Esc)"></button>

                <button class="pswp__button pswp__button--share" style="display:none" title="Share"></button>

                <button class="pswp__button pswp__button--fs" title="Toggle fullscreen"></button>

                <button class="pswp__button pswp__button--zoom" title="Zoom in/out"></button>

                <!-- Preloader demo http://codepen.io/dimsemenov/pen/yyBWoR -->
                <!-- element will get class pswp__preloader--active when preloader is running -->
                <div class="pswp__preloader">
                    <div class="pswp__preloader__icn">
                        <div class="pswp__preloader__cut">
                            <div class="pswp__preloader__donut"></div>
                        </div>
                    </div>
                </div>
            </div>

            <div class="pswp__share-modal pswp__share-modal--hidden pswp__single-tap">
                <div class="pswp__share-tooltip"></div>
            </div>

            <button class="pswp__button pswp__button--arrow--left" title="Previous (arrow left)">
            </button>

            <button class="pswp__button pswp__button--arrow--right" title="Next (arrow right)">
            </button>

            <div class="pswp__caption">
                <div class="pswp__caption__center"></div>
            </div>

        </div>

    </div>

</div>

<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe.min.css">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/default-skin/default-skin.min.css">
<script src="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe-ui-default.min.js"></script>

<script>
    function viewer_init() {
        let pswpElement = document.querySelectorAll('.pswp')[0];
        let $imgArr = document.querySelectorAll(('.article-entry img:not(.reward-img)'))

        $imgArr.forEach(($em, i) => {
            $em.onclick = () => {
                // slider展开状态
                // todo: 这样不好，后面改成状态
                if (document.querySelector('.left-col.show')) return
                let items = []
                $imgArr.forEach(($em2, i2) => {
                    let img = $em2.getAttribute('data-idx', i2)
                    let src = $em2.getAttribute('data-target') || $em2.getAttribute('src')
                    let title = $em2.getAttribute('alt')
                    // 获得原图尺寸
                    const image = new Image()
                    image.src = src
                    items.push({
                        src: src,
                        w: image.width || $em2.width,
                        h: image.height || $em2.height,
                        title: title
                    })
                })
                var gallery = new PhotoSwipe(pswpElement, PhotoSwipeUI_Default, items, {
                    index: parseInt(i)
                });
                gallery.init()
            }
        })
    }
    viewer_init()
</script> 
<!-- MathJax -->

<!-- Katex -->

<!-- busuanzi  -->
 
<script src="/js/busuanzi-2.3.pure.min.js"></script>
 
<!-- ClickLove -->

<!-- ClickBoom1 -->

<!-- ClickBoom2 -->

<!-- CodeCopy -->
 
<link rel="stylesheet" href="/css/clipboard.css">
 <script src="https://cdn.jsdelivr.net/npm/clipboard@2/dist/clipboard.min.js"></script>
<script>
  function wait(callback, seconds) {
    var timelag = null;
    timelag = window.setTimeout(callback, seconds);
  }
  !function (e, t, a) {
    var initCopyCode = function(){
      var copyHtml = '';
      copyHtml += '<button class="btn-copy" data-clipboard-snippet="">';
      copyHtml += '<i class="ri-file-copy-2-line"></i><span>COPY</span>';
      copyHtml += '</button>';
      $(".highlight .code pre").before(copyHtml);
      $(".article pre code").before(copyHtml);
      var clipboard = new ClipboardJS('.btn-copy', {
        target: function(trigger) {
          return trigger.nextElementSibling;
        }
      });
      clipboard.on('success', function(e) {
        let $btn = $(e.trigger);
        $btn.addClass('copied');
        let $icon = $($btn.find('i'));
        $icon.removeClass('ri-file-copy-2-line');
        $icon.addClass('ri-checkbox-circle-line');
        let $span = $($btn.find('span'));
        $span[0].innerText = 'COPIED';
        
        wait(function () { // 等待两秒钟后恢复
          $icon.removeClass('ri-checkbox-circle-line');
          $icon.addClass('ri-file-copy-2-line');
          $span[0].innerText = 'COPY';
        }, 2000);
      });
      clipboard.on('error', function(e) {
        e.clearSelection();
        let $btn = $(e.trigger);
        $btn.addClass('copy-failed');
        let $icon = $($btn.find('i'));
        $icon.removeClass('ri-file-copy-2-line');
        $icon.addClass('ri-time-line');
        let $span = $($btn.find('span'));
        $span[0].innerText = 'COPY FAILED';
        
        wait(function () { // 等待两秒钟后恢复
          $icon.removeClass('ri-time-line');
          $icon.addClass('ri-file-copy-2-line');
          $span[0].innerText = 'COPY';
        }, 2000);
      });
    }
    initCopyCode();
  }(window, document);
</script>
 
<!-- CanvasBackground -->

<script>
  if (window.mermaid) {
    mermaid.initialize({ theme: "forest" });
  }
</script>


    
    

  </div>
</body>

</html>